Optical image stabilization in a scanning folded camera

ABSTRACT

A Tele folded camera operative to compensate for an undesired rotational motion of a handheld electronic device that includes such a camera, wherein the compensation depends on the undesired rotational motion and on a point of view of the Tele folded camera.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a 371 of international application PCT/IB2021/056617filed Jul. 22, 2021, and is related to and claims the benefit ofpriority from U.S. provisional patent application No. 63/064,565 filedAug. 12, 2020, which is incorporated herein by reference in itsentirety.

FIELD

Examples disclosed herein relate in general to digital cameras and inparticular to correction of images obtained with folded digital cameras.

BACKGROUND

Compact digital cameras having folded optics, also referred to as“folded cameras” are known, see e.g. co-owned international patentapplication PCT/IB2016/057366. In handheld electronic devices (alsoreferred to herein as “handheld devices”) such as smartphones, tabletsetc. a folded Tele camera is often part of a multi-camera andaccompanied by one or more additional cameras, e.g. an Ultra-wide cameraand a Wide camera. An Ultra-wide camera has a larger field of view(FOV_(UW)) than a Wide camera, which has a larger FOV_(W) than a Telecamera having FOV_(T) (assuming similar image sensor sizes).

FIG. 1A shows schematically a folded Tele camera numbered 100 from aperspective view. Camera 100 includes a lens 102 with a lens opticalaxis 110, an optical path folding element (OPFE) 104 and an image sensor106. OPFE 104 folds a first optical path along an axis 108 substantiallyparallel to the X axis from an object, scene or panoramic view section114 into a second optical path along an axis 110 substantially parallelto the Z axis. Camera 100 is designed to rotate OPFE 104 around axis 110(the X axis) relative to the image sensor, i.e. in the Y-Z plane, arotation indicated by an arrow 112. That is, folded Tele camera 100 is a“scanning” Tele camera (“STC”). FIG. 1B shows OPFE 104 after rotation by30 degrees from the zero position.

FIG. 1C shows a handheld device 120 including a STC 100 having lens 102,OPFE 104 and image sensor 106 in a top view. A device normal (“N”) isorthogonal to a screen 116 of device 120 and points towards theobserver. The camera's optical axis is parallel to the X axis. In otherexamples, STC 100 may be included in 120 so that the camera's opticalaxis is parallel to the Y axis.

Images are acquired from a certain point of view (POV) of a camera. ThePOV is the direction defined by the vector that has the location of acamera's aperture as starting point and an object point at the center ofthe FOV as end point (see FIG. 3A, with two POV vectors 324 and 328corresponding to two FOV_(T) _(S) 326 and 332). Instead of POV vector,one may also speak of the FOV center direction vector (FOVCD). As anexample, in spherical coordinates (r, θ, φ) defined according to the ISOconvention, the POV for a camera at r=0 is defined by (1, θ, φ), withthe polar angle θ and azimuthal angle φ defining the location of theobject point at the center of the Tele FOV. The length of the POV vectormay be 1 (unit vector) or may have some constant length (e.g. EFL) ormay have a varying length e.g. so that it comes to lie on a specificplane.

As e.g. described in the co-owned PCT Patent Application No. PCT/IB2016/057366 and with reference to FIGS. 1A-1B, rotation of the OPFEmay be performed around the X axis and around the Y axis for “scanning”with the FOV in 2 dimensions (2D) in FIGS. 1A-1B.

Modern cameras that are included in handheld devices often includeoptical image stabilization (OIS) for mitigating undesired camera motioncaused by a user's hand motion (often referred to as “hand-shake”). ForOIS, optical components are moved to reduce movements of imaged objectson a camera's image sensor. The lens module and/or the image sensorand/or the OPFE and/or the entire camera can be moved. An inertialmeasurement unit (IMU) included in the handheld device provides motiondata along 6 degrees of freedom, namely and with reference to FIG. 1C,linear movements in X-Y-Z, roll “tilt about” (or “tilt around”) the Zaxis, yaw (tilt around the Y axis) and pitch (tilt around the X axis).Usually, OIS is provided for Pitch and Yaw rotation compensation only,and not for roll rotation, as Pitch and Yaw rotation account for themajor share of image deterioration caused by hand-shake. Coordinatesystems of the IMU, of a regular (i.e. a non-scanning) camera and of theincluding handheld device can be aligned and do not evolve in time. Fora STC, this is not valid. The relation between a handheld device'scoordinate system and that of a STC does change when FOV scanning isperformed. Therefore, OIS as known in the art cannot be used for handmotion compensation in a STC.

There is a need for and it would be advantageous to have OIS forscanning Tele cameras.

SUMMARY

Henceforth and for simplicity, the terms “electronic device”,“electronic handheld device” “handheld device” or just “device” are usedinterchangeably. Henceforth and for simplicity, the term “smartphone”may be used to represent all electronic handheld devices having scanningfolded cameras and implementing methods for OIS in such camerasdescribed herein.

In various embodiments, there are provided Tele folded cameras operativeto compensate for an undesired rotational motion of a handheldelectronic device that includes such a camera, wherein the compensationdepends on the undesired rotational motion and on a point of view of theTele folded camera.

In various embodiments, a handheld electronic device comprises: a Telefolded camera comprising an OPFE for folding light from a first opticalpath that forms an angle of less than 90 degrees to a normal of thedevice toward a second optical path substantially orthogonal to thenormal of the device, a lens with a lens optical axis along the secondoptical path, and an image sensor, wherein the device is a handheldelectronic device; an OPFE actuator for tilting the OPFE in one or moredirections to direct a point of view (POV) of the Tele folded cameratowards a segment of a scene; a motion sensor for sensing an undesiredrotational motion of the device; and

at least one actuator for moving at least one component of the Telefolded camera to compensate for the undesired rotational motion of thedevice, wherein the compensation depends on the undesired rotationalmotion of the device and on the Tele folded camera POV.

In some embodiments, the undesired rotation motion is around the devicenormal.

In some embodiments, a device further comprises a Wide camera having afield of view FOVw larger than a field of view FOV_(T) of the Telecamera.

In some embodiments, the sensing the rotational motion includesmeasuring the rotation motion in three directions.

In some embodiments, the actuator for moving the component of the Telefolded camera to compensate for the device's undesired rotational motionis the OPFE actuator for tilting the OPFE in one or more directions todirect a point of view (POV) of the Tele folded camera towards a segmentof a scene.

In some embodiments, the moving of the component of the Tele foldedcamera to compensate for the device's undesired rotational motionincludes moving the lens.

In some embodiments, the moving of the component of the Tele foldedcamera to compensate for the device's undesired rotational motionincludes moving the image sensor.

In some embodiments, a device further comprises a processing unitconfigured to perform a coordinate transformation to align coordinatesof the Tele camera with coordinates of the handheld device or viceversa.

In some embodiments, a device further comprises a processing unitconfigured to perform a coordinate transformation that alignscoordinates of a reference coordinate system with coordinates of thehandheld device and coordinates of the Tele camera.

In some embodiments, the coordinate transformation is performed usingRodrigues' rotation formula.

In some embodiments, the motion sensor includes an inertial measurementunit (IMU).

In some embodiments, a device further comprises a microcontroller unit(MCU) configured to read out the motion sensor and to provide controlsignal to the rotational motion compensation actuator. In someembodiments, the MCU is included in an application processor (AP).

In some embodiments, a device further comprises an application processorconfigured to provide POV control signal to the OPFE actuator fortilting the OPFE.

In various embodiments, there are provided methods comprising: providinga handheld device comprising a Tele folded camera that includes an OPFEfor folding light from a first optical axis that forms an angle of lessthan 90 degrees to a normal of the device toward a second optical axissubstantially orthogonal to a normal of the device, a lens with a lensaxis along the second optical axis, and an image sensor; providing anOPFE actuator for tilting the OPFE in one or more directions to direct apoint of view (POV) of the Tele folded camera towards a segment of ascene; sensing an undesired rotational motion of the device; andcompensating for the undesired rotational motion, wherein thecompensation depends on the undesired rotational motion and on the Telefolded camera's POV.

In some embodiments, the compensating for the undesired rotationalmotion includes moving a component of the Tele folded camera.

In some embodiments, the compensating for the undesired rotationalmotion includes compensating for a rotational motion around the device'snormal direction.

In some embodiments, a method further comprises performing a coordinatetransformation to align coordinates of the Tele camera with coordinatesof an IMU.

In some embodiments, a method further comprises performing a coordinatetransformation to coordinates of the IMU with coordinates of the Telecamera.

In some embodiments, a method further comprises performing a coordinatetransformation to align coordinates of a reference coordinate systemwith coordinates of the IMU and coordinates of the Tele camera.

In some embodiments, the performing the coordinate transformationincludes performing the transformation using Rodrigues' rotationformula.

In some embodiments, the sensing an undesired rotational motion of thedevice includes sensing the undesired rotational motion in threedirections.

In some embodiments, the compensating for the undesired rotationalmotion of the device includes rotating the OPFE.

In some embodiments, the compensating for the undesired rotationalmotion of the device includes moving the lens.

In some embodiments, the compensating for the undesired rotationalmotion of the device includes moving the image sensor.

In some embodiments, the compensating for the undesired rotationalmotion includes calculating a changed POV caused by the undesiredrotational motion in the X direction by using the equation: P^(F)_(P)=(P^(I)·cos(hnd_pitch)+cross(P^(I),R_(P))·sin(hnd_pitch)+R_(P)·(dot(P^(I), R_(P)) (1-cos (hnd_pitch)))).

In some embodiments, the compensating for the undesired rotationalmotion includes calculating a changed POV caused by the undesiredrotational motion in the Y direction by using the equation: P^(F)_(Y)=(P^(I)·cos(hnd_yaw)+cross(P^(I),R_(Y))·sin(hnd_yaw)+R_(Y)·(dot(P^(I), R_(Y)).(1-cos (hnd_yaw)))).

In some embodiments, the compensating for the undesired rotationalmotion includes calculating a changed POVcaused by the undesiredrotational motion in the X direction by using the equation: P^(F)_(R)=(P^(I)·cos(hnd_roll)+cross(P^(I),R_(R))·sin(hnd_roll)+R_(R)·(dot(P^(I), R_(R)).(1-cos (hnd_roll)))).

In some embodiments, the compensating for the undesired rotationalmotion includes calculating a direction of a changed POV caused by theundesired rotational motion in X, Y and Z direction together by usingthe equation: P^(F),=P^(I)+(P^(I)-P^(F) _(P))+(P^(I)-P^(F)_(Y))+(P^(I)-P^(F) _(R)).

In some embodiments, the compensating for the undesired rotationalmotion includes calculating a vector of a changed POV caused by by theundesired rotational motion in X, Y and Z direction together by usingthe equation: P^(F)′P^(F)′.EFL_(T)/P^(F)′_(z).

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of embodiments disclosed herein are describedbelow with reference to figures attached hereto that are listedfollowing this paragraph. The drawings and descriptions are meant toilluminate and clarify embodiments disclosed herein, and should not beconsidered limiting in any way. Like elements in different drawings maybe indicated like numerals.

FIG. 1A shows schematically a known folded scanning camera from aperspective view;

FIG. 1B shows the OPFE in the Tele camera of FIG. 1A after rotation by30 degrees from a zero position;

FIG. 1C shows a scanning camera such as shown in FIGS. 1A-B integratedas a “rear” or “world-facing” camera in a smartphone;

FIG. 2A shows exemplarily a smartphone including a first, scanning Telecamera at a zero position, as well as a second, Wide camera;

FIG. 2B shows the smartphone of FIG. 2A with the Tele camera at anon-zero position;

FIG. 2C shows the smartphone of FIG. 2A with the Tele camera at anothernon-zero position;

FIG. 3A shows a 2-dimensional (2D) chart used to derive a coordinatesystem for the Tele camera;

FIG. 3B shows impact of rotational device motion caused by hand shake onthe 2D chart of FIG. 3A;

FIG. 3C shows in a flow chart main steps of a method for scanning Telecamera OIS disclosed herein;

FIG. 4A shows schematically in a block diagram an embodiment of ahandheld device that includes multi-aperture cameras with at least onescanning Tele camera disclosed herein;

FIG. 4B shows schematically in a block diagram another embodiment of ahandheld device that includes multi-aperture cameras with at least onescanning Tele camera disclosed herein.

DETAILED DESCRIPTION

FIG. 2A shows exemplarily a smartphone 200 comprising a STC 202 at azero position, and a Wide camera 204. Wide camera 204 is not a scanningcamera and its POV (“POVw”) is parallel to a device normal N (parallelto Z-axis) of the smartphone. Device normal N is parallel (oranti-parallel) to a normal onto a surface of smartphone 200 that has thelargest area. A coordinate system of the IMU of smartphone 200 (such asIMU 460 in FIGS. 4A and 4B, not shown here) may be aligned with acoordinate system of smartphone 200 such as the coordinate system shownin FIG. 2A, where the three axes of the coordinate system are parallelto the three symmetry axes of smartphone 200, so that the Z axis of theIMU's (and smartphone 200's) coordinate system is parallel to POV_(W).The POV of STC 202 (“POV_(T)”) is directed its zero position(“POV_(T,0)”), corresponding to an OPFE rotation state such as shown inFIG. 1A. With POV_(T) at zero position, the coordinate systems of IMU,Wide camera 204 and STC 202 align.

In a first exemplary method for OIS (“Example 1”), consider OIS for Widecamera 204 that (for the sake of simplicity) may correct for pitchrotation only. For detecting the amount of undesired hand motion, onecould read out the value for pitch rotation around the X axis from theIMU (“X_(IMU)”) and move e.g. the lens in one particular direction(dir₁) by a particular amount, wherein the amount (or stroke) ofmovement is proportional to X_(IMU), i.e. the lens stroke S_(W) fulfillsS_(W)=C_(W).X_(IMU) (with some constant C_(W)). The same holds for OISof STC 202 at zero position. By moving the lens by S_(T)=C_(T).X_(IMU)(with some constant C_(T)) in din the hand motion is compensated.

FIG. 2B shows smartphone 200 with STC 202 at a non-zero position.POV_(T) has an angle of α degrees with respect to POV_(w). For example,for α=30 degrees, this corresponds to an OPFE rotation state such asshown in FIG. 1B. The coordinate systems of IMU, Wide camera 204 and STC202 do not align anymore.

Consider Example 1 (hand motion in pitch direction) with STC 202 at anon-zero position. OIS for Wide camera 204 may be performed as inExample 1. However, for OIS of STC 202, the method of Example 1 does notallow to perform hand motion compensation anymore, i.e. there is (ingeneral) no C_(T) so that by moving the Tele lens byS_(T)=C_(T).X_(IMU), the hand motion is compensated. This is because thecoordinate systems of STC 202 and the IMU are not aligned anymore.

For a second exemplary method for OIS (“Example 2”), refer to FIG. 2C.Compared to FIG. 2A, POV_(T) is rotated by 90 degree around the Y axis,i.e. POV_(T) and POV_(W) are perpendicular to each other. As in Example1, we consider OIS for the Wide camera for correction of pitch rotationonly. Hand motion can be fully compensated by reading the IMU's valuefor rotation X_(IMU) and by moving a lens of the Wide camera (not shown)by S_(W)=C_(W).X_(IMU) (with some constant C_(W)) in dir₁. However, thehand motion cannot be compensated by moving a lens of the STC (notshown, but similar to lens 102) by S_(T)=C_(T).X_(IMU) (with someconstant C_(T)) in dir₁. Instead, the rotation direction must bemodified from din to a particular direction dir₂ which is different fromdir₁. The hand motion can be compensated by moving the STC lens byS_(T)=C_(T).X_(IMU) in dir₂. In general, for a STC the OIS axes dependon the POV or scanning state of the STC and are thus not constant, as itis the case for a Wide camera.

FIG. 3A shows a 2-dimensional (2D) chart 320 for deriving a coordinatesystem for a STC.

An aperture 322 of the STC is located at coordinates (0, 0, 0). A zerostate STC POV_(T) (POV_(T,0)) 324 corresponds to a first optical pathwhich is parallel to a device normal N (see FIG. 2A-C) and may have thecoordinates (0, 0, EFL_(T)), with EFL_(T) being the EFL of the STC.FOV_(T) 326 corresponds to the FOV_(T) of the STC at POV_(T,0) 324. Adesired or target POV_(T) 328 (“POV_(T,T)”) with corresponding FOV_(T)332 is shown as well.

FIG. 3B shows 2D chart 320 of FIG. 3A after the handheld device thatincludes the STC underwent a rotational “roll” motion around the Z axis,e.g. because of a user's hand motion. POV_(T,0) 324 did not undergo anychange. However, the corresponding FOV_(T) changed to a rotated FOV_(T)326′. In contrast, the rotational motion changed POV_(T,T) 328 toPOV_(T,T) 328′. The change of a POV such as POV_(T,T) 328 in response toa rotational device motion depends not only on the angle or amount ofrotation, but also on the position of POV_(T).

FIG. 3C shows in a flow chart main steps of a method for STC OISdisclosed herein.

In a first step 302, a command triggered by a human user or by a programand processed by a FOV scanner 442 (FIG. 4A) directs FOV_(T) to a regionof interest (ROI) within a scene. The scanning may be performed byrotating an OPFE with an OPFE actuator 414 (FIG. 4A). The FOV scanningby OPFE rotation is not performed instantaneously, but requires somesettling time, which may be about 1-50 ms for scanning 2-5 degrees andabout 5-500 ms for scanning 10-45 degrees. After the settling time, theSTC is operational for capturing Tele images. The STC may be focused toan object by a user command or autonomously. The STC's scanningdirection may be given by an initial (or target) POV vector P^(I). Instep 304, the IMU is read out and provides rotational movements aroundthe Pitch, Yaw and Roll directions, i.e. X^(IMU), Y^(IMU) and Z^(IMU)respectively. Usually, IMU provides data on the angular accelerationwhich is to be integrated for determining the rotation angle. The IMUdata may be used to calculate the undesired rotational motion of thedevice.

In step 306, a coordinate transformation is performed. The coordinatetransformation is required because the STC's POV change caused by anundesired rotational motion of the device and the sensing of theundesired rotational motion occur in different coordinate systems.

A processing unit such as an AP or a MCU may be configured forperforming the coordinate transformation (e.g. AP 440 of device 400 ordevice 480, or MCU 470 of device 400 in FIG. 4A). In some examples, anAP or MCU may solve the below equations analytically, or AP or MCU mayuse a polynomial fit or a linear fit for solving the equationsapproximately. In other examples, the AP or MCU may not performcalculations but use a Look Up Table (LUT) for coordinatetransformation. In some examples and such as e.g. shown in FIG. 4A, thecoordinate transformation may be performed by a MCU such as MCU 470connected to the STC module 410.

In some examples, the transformation may be performed in order toexpress the coordinates of the STC in the coordinate system of the IMU.Device rotations and compensation motions may then be calculated in theIMU's coordinate system.

In some examples, a 2D chart such as chart 320 shown in FIG. 3B may beused to express the coordinates of the STC in the IMU's coordinatesystem. Chart 320 may resemble a calibration chart for calibrating theSTC or for calibrating a dual-camera, e.g. including a Tele camera and aWide camera. STC aperture 322 may be located at (0, 0, 0). The handhelddevice may be pointed towards chart 320 in “landscape” direction, i.e.with reference to the coordinate system of FIG. 3B, the long side of asmartphone as shown in FIG. 1C may be parallel to the X axis and theshort side parallel to the Y axis, with the STC aperture pointingtowards the chart in Z direction. All POVs that the STC can reach aregiven by “POV vectors” or “camera pointing vector” P which are pointingto coordinates lying on chart 320. The coordinates of the zero stateposition may be (0, 0, EFL_(T)) with EFLT being the EFL of the STC. Atzero position, the coordinates of the IMU (and of the handheld device)overlap with the STC's coordinates.

If the STC is directed to a non-zero POV, a coordinate transform fromthe IMU's to the STC's coordinates must be performed. In some examples,Rodrigues' rotation formula may be used. The IMU's pitch/yaw/rollrotation values may be named “hnd_pitch”, “hnd_yaw” and “hnd_roll”. IMUprovides hnd_pitch, hnd_yaw and hnd_roll in a coordinate system havingthe following unit vectors:

-   -   Pitch unit vector R_(P): R_(P)=(1, 0, 0),    -   Yaw unit vector R_(Y): R_(Y)=(0, 1, 0),    -   Roll unit vector R_(R): R_(R)=(0, 0, 1).

In general, OIS corrects small angles only. Therefore, in somesituations and approximately, one may treat the pitch/yaw/roll rotationsindependently. For any (slight) rotation of the device, Rodrigues'rotation formula may be applied to pitch/yaw/roll rotationsindependently, wherein the (slight) rotation may be represented by thesum over the pitch/yaw/roll rotations. A hand motion only by hnd_pitch,or only by hnd_yaw or only by hnd_roll (in the IMU's coordinates R^(P),R^(Y) and R^(R)) applied to any initial POV vector P^(I) may result inthe following final POV vector P^(F) (“cross(x, y)” indicates the crossproduct of vectors x and y, “dot(x, y)” indicates the dot product ofvectors x and y):

POV vector P^(F) _(P) after rotation by hnd_pitch around R_(P) (hnd_yaw,hnd_roll=0): P^(F) _(P)=(P^(I)·cos(hnd_pitch)+cross(P^(I),R^(P))·sin(hnd_pitch)+R^(P)·(dot(P^(I), R^(P)).(1-cos(hnd_pitch))));

POV vector P^(F) _(Y) after rotation by hnd_yaw around R_(Y) (hnd_pitch,hnd_roll=0): P^(F) _(Y)=(P^(I)·cos(hnd_yaw)+cross(P^(I),R^(Y))·sin(hnd_yaw)+R^(Y)(dot(P^(I), R^(Y)).(1-cos(hnd_yaw))));

POV vector P^(F) _(R) after rotation by hnd_roll around R_(R)(hnd_pitch,hnd_yaw=0): P^(F) _(R)=(P^(I)·cos(hnd_roll)+cross(P^(I),R^(R))·sin(hnd_roll)+R^(R)(dot(P^(I), R^(R)).(1-cos(hnd_roll)))).

For small angles, a final POV vector (before normalization) P^(F)′ thatunderwent both Pitch, Yaw and Roll rotations may be given by:

P^(F)′=P^(I)+(P^(I)-P^(F) _(P))+(P^(I)-P^(F) _(Y))+(P^(I)-P^(F) _(R))

A normalization may be performed in order to ensure that the final POVvector P^(F) comes to lie on chart 320. In some examples, P^(F) may beobtained by normalizing P^(F)′ with EFL_(T)/P^(F)′_(z), whereinP^(F)′_(z) is the z-component of P^(F)′, i.e.:

P^(F)=P^(F)′·EFL_(T)/P^(F)′_(z).

From the above equations it is evident that for compensating forundesired rotational hand motion in a STC, in contrast for anon-scanning camera like e.g. Wide camera 204, where one may compensatethe undesired rotational hand motion around yaw and pitch only, one mustcompensate rotational hand motion around the three directions yaw, pitchand roll.

In other examples for coordinate transformation, the transformation maybe performed to express the coordinates of the IMU in the coordinatesystem of the STC. Hand motion rotations and compensation motions maythen be calculated in the STC's coordinate system. As above, Rodrigues'rotation formula may be used.

In yet other examples for coordinate transformation, the transformationmay be to a third coordinate system (“reference system”). Both thecoordinates of the STC and of the IMU are expressed in the referencecoordinate system. Hand motion rotations and compensation motions maythen be calculated in the reference coordinate system. As above,Rodrigues' rotation formula may be used.

In step 308, movement for OIS may be performed. In some examples, OISmay be performed by moving the STC's OPFE. In other examples, a lenssuch as lens 102 and/or an image sensor such as image sensor 106 may bemoved for OIS. Assuming ideal OIS, the movement of OPFE and/or lensand/or sensor may lead to a POC vector modification Pois that exactlycancels the effect of the hand motion onto the POV vector, i.e.:P^(F)+P^(OIS)=P^(I). So after performing step 308 the STC is againdirected towards P^(I). In other examples, the entire STC may be movedfor OIS. I.e. OPFE, lens and image sensor are moved together as one unitfor OIS.

In some embodiments, steps 304-308 may be repeated for stabilizing theSTC continuously. The OIS cycles that include steps 304-308 may beperformed at frequencies of e.g. 500 Hz-100 kHz. STC images or imagestreams are captured while the above OIS method is performed.

In some embodiments, an IMU may be fixedly attached to the OPFE, so thatwhen moving the OPFE, the IMU moves accordingly, too. This allows forusing coordinate systems having identical basis vectors for both the STCand the IMU, so that the coordinate transform of step 306 is notrequired.

In some embodiments, a sensor actuator may actuate the image sensor forcorrecting POV aberrations of a STC image. As described in the co-ownedinternational patent application PCT/IB2021/056311, a STC imageundergoes POV aberrations. One aberration is a rotation of the STC imageon the image sensor (“rotational POV aberration”). When an undesiredrotational hand motion is compensated by moving an OPFE as disclosedherein, the moving of the OPFE introduces a POV aberration. A sensoractuator may be used to rotate an image sensor around a normal of theimage sensor for compensating the rotational POV aberration.

FIG. 4A shows schematically an embodiment of a handheld device numbered400 and including multi-aperture cameras with at least one STC disclosedherein. Device 400 comprises a STC module 410 that includes an OPFE 412as well as an OPFE actuator 414 for FOV scanning and/or OIS, and a Telelens module 420 that forms a Tele image recorded by an image sensor 416.A Tele lens actuator 422 may move lens module 420 for focusing and/orOIS. Handheld device 400 may further comprise an application processor(AP) 440 that includes a FOV scanner 442, an OIS controller 444, animage generator 446 and an object tracker 448.

In other examples, device 400 may comprise a STC that includes two OPFEsas well as an OPFE actuator for each of the two OPFEs. In some examples,the OPFE actuators may actuate the OPFEs for performing OIS as disclosedherein. In other examples, a lens actuator may actuate a lens or asensor actuator may actuate a sensor for performing OIS as disclosedherein. A STC camera based on two OPFEs is described for example inPCT/IB2021/054186. In such a STC, the optical path within the camera isfolded twice, so that one speaks of a double-folded scanning Telecamera.

Handheld device 400 further comprises a Wide (or Ultra-Wide) cameramodule 430 which includes a second lens module 434 that forms an imagerecorded by a second image sensor 432. A second lens actuator 436 maymove lens module 434 for focusing and/or OIS. In some examples, the STCcan scan the entire FOV_(W) or an even larger FOV. In other examples,the STC can scan a FOV that is smaller than FOV_(W).

In some examples, object tracker 448 may be configured to track anobject in FOV_(W) and provide tracking data to FOV scanner 442 and/orthe OIS controller 444. Based on the tracking data, FOV scanner 442and/or the OIS controller 444 may provide control signals to OPFEactuator 414 which actuate an OPFE rotation for tracking an object withthe STC. As an example, one may track an object so that it centers atthe center of FOV_(T). Examples 3-7 described below refer to thistracking scenario, where the Wide camera image data is used to providetracking information which triggers Tele FOV scanning.

In some examples, tracking information and OIS information may interfereand coordination between tracking and OIS may be required for achievinga desired object tracking and/or OIS outcome.

As a third exemplary method for OIS, consider a device such as device400 or 480 including a Wide camera and a STC both not having OIS. TheSTC may track an object at rest so that the object's center is locatedat the center of FOV_(T). The tracking may occur in real-time (RT), i.e.we assume that there is no delay between the detection of a trackingdeviation and its compensation. A device's rotational motion caused by auser's hand motion will be detected as an object movement in the Widecamera. In response, a tracking movement of the STC will be triggeredand the object's location in the Tele FOV will be updated. Inconclusion, in the RT scenario the object tracker performs OIS in asense that the object will always be located in the center of FOV_(T)and will not be affected from hand motion of a user.

As a fourth exemplary method for OIS, consider a device such as device400 or 480 including a Wide camera not having OIS and a STC having OIS.As in example 3, we assume RT object tracking on FOV_(W) so that a(non-moving) object's center is located at the center of FOV_(T). OISmay be performed in RT as well. A device's rotational motion caused by auser's hand motion will be detected as an object movement in the Widecamera. In response, a tracking movement ΔT for the STC will betriggered. Simultaneously, the device's rotational motion will also bedetected by the STC's OIS and an OIS movement ΔOIS of the STC will betriggered in response. OIS movement may be performed according the OISmethod disclosed herein. ΔT and ΔOIS are identical in terms of directionand magnitude, i.e. a STC movement of 2·ΔT=2·ΔOIS will be triggered,which is double the amount of movement required (i) for keeping theobject at the center of FOV_(T) (desired tracking outcome) and (ii) forsuppressing the impact of hand motion on the STC image (desired OISoutcome). In conclusion, the desired outcome is not achieved for eitherTele tracking or Tele OIS. Therefore, in some examples, the STC's OIS isdisabled when using object tracking.

As a fifth exemplary method for OIS, consider a device such as device400 or 480 including a Wide camera not having OIS and a STC having OIS.Object tracking may be performed on FOV_(W) so that a (non-moving)object's center is located at the center of FOV_(T). However, Objecttracking and OIS may not be performed in RT. In general, OIS isperformed at higher frequencies than object tracking. As an example, OISmay be performed at 500 Hz-100 kHz and object tracking may be performedat 1 Hz-100 Hz. In some examples, for preventing undesired interferencebetween OIS and object tracking, one may disable OIS when using objecttracking. In other embodiments, one may separate control of OIS andobject tracking in the frequency domain. As an example, for device'srotational motion caused by a user's hand motion that occurs at afrequency higher than e.g. 30 Hz, one may use OIS for device motioncorrection. For frequencies lower than e.g. 30 Hz one may not use OISfor device motion correction. Instead the low frequency device motionwill be compensated by the object tracker.

As a sixth exemplary method for OIS, consider a device such as device400 or 480 including a Wide camera having OIS and a STC not having OIS.Object tracking may be performed on FOV_(W) so that a (non-moving)object's center is located at the center of FOV_(T). Object tracking andOIS may be performed in RT. As of the Wide camera's OIS, a device'srotational motion caused by a user's hand motion will have no impact onthe Wide image stream. As the object does not move in FOV_(W), notracking movement of the STC will be triggered. In conclusion, there isno hand motion compensation and the object will not be located at thecenter of FOV_(T) anymore, leading to an undesired object trackingoutcome. In some examples for preventing this undesired outcome, one maydisable the Wide camera's OIS when performing object tracking. In otherexamples, object tracking control signals that are supplied to the STCmay additionally include the Wide camera's OIS control signals. Bysuperimposing the two signals, the benefits of both Wide camera OIS andproper STC tracking may be enjoyed.

As a seventh exemplary method for OIS, consider a device such as device400 or 480 with both the Wide camera and the STC having OIS. We assumeRT tracking so that an object's center is located at the center ofFOV_(T). A device's rotational motion caused by a user's hand motionwill be corrected by an OIS movement in both the Wide camera and the STCin RT. In conclusion, a user's hand motion will not impact the desiredoutput of the object tracker.

Calibration data may be stored in a first memory 424, e.g. in an EEPROM(electrically erasable programmable read only memory) and/or in a secondmemory 438 and/or in a third memory 450 such as a NVM (non-volatilememory). The calibration data may comprise calibration data between Widecamera 430 and STC 410. The calibration data may further comprisecalibration data between an OPFE's position and the STC's correspondingPOV.

Handheld device 400 further comprises an inertial measurement unit (IMU,for example a gyroscope) 460 that supplies motion information of 400.For example, a microcontroller unit (MCU) 470 may be used to read andprocess data of IMU 460. In some examples, the MCU may be controlled byan OIS controller 444 which is part of AP 440. In some examples, step304 and step 306 may be performed by the MCU and step 308 may beperformed by OPFE actuator 414 (and/or lens actuator 436 and/or sensoractuator 418 in case OIS is performed by lens shift or sensor shiftrespectively). In some examples, MCU 470 may be integrated into AP 440.

Another embodiment of a handheld device numbered 480 and comprising amulti-aperture camera with at least one STC as disclosed herein is shownin FIG. 4B. An MCU (not shown) for reading and processing motion datafrom IMU 460 and for supplying OIS control signals may be included intoSTC module 410, e.g. into the driver of OPFE actuator 414.

In some examples, additional data may be used for hand motionestimation. Additional data may e.g. be image data from the Wide camera430 or data from additional sensing units present in the handhelddevice.

In some examples, image data from Wide camera 430 may be used toestimate an “optical flow” from a plurality of images as known in theart, wherein OIS controller 444 may use the data of the optical flowtogether with data from IMU 460 for estimating motion of device 400. Inother examples, only optical flow data estimated from image data ofcamera 410 and/or camera 430 may be used for estimating motion of device400.

Image generator 446 may be configured to generate images and imagestreams. In some examples, image generator 446 may be configured to useonly first image data from camera 430. In other examples, imagegenerator 446 may use image data from camera 410 and/or camera 430.

While this disclosure has been described in terms of certain embodimentsand generally associated methods, alterations and permutations of theembodiments and methods will be apparent to those skilled in the art.The disclosure is to be understood as not limited by the specificembodiments described herein, but only by the scope of the appendedclaims.

Unless otherwise stated, the use of the expression “and/or” between thelast two members of a list of options for selection indicates that aselection of one or more of the listed options is appropriate and may bemade.

It should be understood that where the claims or specification refer to“a” or “an” element, such reference is not to be construed as therebeing only one of that element.

Furthermore, for the sake of clarity the term “substantially” is usedherein to imply the possibility of variations in values within anacceptable range. According to one example, the term “substantially”used herein should be interpreted to imply possible variation of up to5% over or under any specified value. According to another example, theterm “substantially” used herein should be interpreted to imply possiblevariation of up to 2.5% over or under any specified value.

According to a further example, the term “substantially” used hereinshould be interpreted to imply possible variation of up to 1% over orunder any specified value.

All patents and/or patent applications mentioned in this specificationare herein incorporated in their entirety by reference into thespecification, to the same extent as if each individual reference wasspecifically and individually indicated to be incorporated herein byreference. In addition, citation or identification of any reference inthis application shall not be construed as an admission that suchreference is available as prior art to the present invention.

1. A device, comprising: a Tele folded camera comprising an optical pathfolding element (OPFE) for folding light from a first optical path thatforms an angle of less than 90 degrees to a normal of the device towarda second optical path substantially orthogonal to the normal of thedevice, a lens with a lens optical axis along the second optical path,and an image sensor, wherein the device is a handheld electronic device;an OPFE actuator for tilting the OPFE in one or more directions todirect a point of view (POV) of the Tele folded camera towards a segmentof a scene; a motion sensor for sensing an undesired rotational motionof the device; at least one actuator for moving at least one componentof the Tele folded camera to compensate for the undesired rotationalmotion of the device, wherein the compensation depends on the undesiredrotational motion of the device and on the Tele folded camera POV, and aprocessing unit configured to perform a coordinate transformation toalign coordinates of the Tele folded camera with coordinates of thedevice or vice versa.
 2. The device of claim 1, wherein the undesiredrotation motion is around the device normal.
 3. The device of claim 1,further comprising a Wide camera having a field of view FOVw larger thana field of view FOV_(T) of the Tele camera.
 4. The device of claim 1,wherein the sensing of the undesired rotational motion includes sensingof the undesired rotational motion in three directions.
 5. The device ofclaim 1, wherein the compensating of the undesired rotational motionincludes compensating the undesired rotational motion in threedirections.
 6. The device of claim 1, wherein the at least one actuatorfor moving the at least one component of the Tele folded camera tocompensate for the device undesired rotational motion includes an OPFEactuator that moves the OPFE.
 7. The device of claim 1, wherein the atleast one actuator for moving the at least one component of the Telefolded camera to compensate for the device undesired rotational motionincludes a lens actuator that moves the lens.
 8. The device of claim 1,wherein the at least one actuator for moving the at least one componentof the Tele folded camera to compensate for the device undesiredrotational motion includes a sensor actuator that moves the sensor. 9.(canceled)
 10. A device, comprising: a Tele folded camera comprising anoptical path folding element (OPFE) for folding light from a firstoptical path that forms an angle of less than 90 degrees to a normal ofthe device toward a second optical path substantially orthogonal to thenormal of the device, a lens with a lens optical axis along the secondoptical path, and an image sensor, wherein the device is a handheldelectronic device: an OPFE actuator for tilting the OPFE in one or moredirections to direct a point of view (POV) of the Tele folded cameratowards a segment of a scene; a motion sensor for sensing an undesiredrotational motion of the device; at least one actuator for moving atleast one component of the Tele folded camera to compensate for theundesired rotational motion of the device, wherein the compensationdepends on the undesired rotational motion of the device and on the Telefolded camera POV; and a processing unit configured to perform acoordinate transformation that aligns coordinates of a referencecoordinate system with coordinates of the device and coordinates of theTele folded camera.
 11. The device of claim 1, wherein the coordinatetransformation is performed using Rodrigues' rotation formula.
 12. Thedevice of claim 1, wherein the coordinate transformation is performed byan analytical solution.
 13. The device of claim 1, wherein thecoordinate transformation is performed by an approximate solution. 14.The device of claim 1, wherein the motion sensor includes an inertialmeasurement unit (IMU).
 15. The device of claim 1, further comprising amicrocontroller unit (MCU) configured to read out the motion sensor andto provide a control signal to the at least one actuator for moving theat least one component of the Tele folded camera to compensate for theundesired rotational motion.
 16. The device of claim 15, wherein the MCUis included in an application processor.
 17. The device of claim 1,further comprising an application processor configured to provide POVcontrol signals to the OPFE actuator for tilting the OPFE.
 18. Thedevice of claim 1, wherein the Tele folded camera is a double-foldedTele camera comprising two OPFEs.
 19. The device of claim 3, whereinWide image data is used to track an object in FOV_(W) and wherein thetracking information is used to direct the POV of the Tele folded cameratowards the tracked object for object tracking with the Tele foldedcamera.
 20. The device of claim 19, wherein the moving of a component ofthe Tele folded camera to compensate for the undesired rotational motionof the device is disabled during the object tracking with the Telefolded camera.
 21. The device of claim 19, wherein the Wide cameraadditionally includes a component which is moved to compensate for theundesired rotational motion of the device, and wherein the moving of theWide camera component is disabled during the object tracking with theTele folded camera.
 22. The device of claim 19, wherein the moving acomponent of the Tele folded camera to compensate for the undesiredrotational motion of the device is performed at a frequency rangedifferent from a frequency range that is used for the object trackingwith the Tele folded camera.
 23. The device of claim 19, wherein afrequency range <30 Hz is used for the object tracking with the Telefolded camera, and wherein a frequency range >30 Hz is used tocompensate for the undesired rotational motion of the device.
 24. Thedevice of claim 19, wherein a frequency range <100 Hz is used for theobject tracking with the Tele folded camera, and wherein a frequencyrange >200 Hz is used to compensate for the undesired rotational motionof the device. 25-42. (canceled)